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ABSTRACT 


This  paper  analyzes  two  work  processes 
involved  in  defense  acquisition  vhich  are 
replete  with  uncertainties.  These  are  the 
proposal  phase  and  the  architectural  design 
phase.  Both  phases  involve  a vendor  design- 
ing alternative  system  in  response  to  a set 
of  stated  (and  perceived.)  requirements, 
followed  by  the  government  agency's  final 
selection  of  a preferred  system  design  (and  a 
preferred  contractor).  Since  these  efforts 
many  times  occur  during  the  preliminary 
phases  of  the  acquisition  process,  many 
uncertainties  are  present,  including:  tech- 

nological uncertainties,  uncertainties  in  the 
timely  availability  . ' inputs,  workload 
uncertainties,  and  equipment  reliability  and 
maintainability,  all  of  which  lead  to  per- 
formance, schedule,  and  cost  uncertainties. 

Several  issues  involving  these  uncertainties 
are  identified: 

1)  Do  government  agencies  provide  vendors 
with  sufficient  information  to  enable 
them  to  design  their  most  cost-effective 
systems  with  respect  to  these  uncertain- 
ties? 

2)  What  additional  information  should  be 
provided  which  will  enable  vandors  to  do 
so? 

3)  What  credible  evidence  should  vendors 
provide  in  their  proposals  and  system 
designs  which  can  increase  the  govern- 
ment's confidence  that  the  system  being 
proposed  will  in  fact  be  delivered  within 
the  schedule  and  cost  estimated? 

Finally,  a systems  evaluation  methodology  is 
described  and  illustrated,  providing  a 
recommended  way  of  dealing  with  these  issues. 


OVERVIEW 

Defense  systems  ara  composed  of  elements 
which  inherently  involve  various  uncertain- 
ties, including  technological  uncertainties, 
traneportation  uncertainties,  equipment  un- 


reliability. In  general,  we  know  how  to  deal 
with  these  factors.  This  paper  focuses  on 
certain  deficiencies  in  the  systems 

acquisition  process  itself,  which  prevent  the 
government  frcm  obtaining  the  most  cost- 
effective  system  meeting  the  needs  and  con- 
straints. The  paper  also  presents  a method 
of  overcoming  these  deficiencies. 

The  specific  parts  of  the  system  acquisition 
process  to  be  focused  upon  are: 

o The  proposal  phase  from  RFP  to  Source 
Selection 

o The  architectural  design  phase  in 
which  a preferred  preliminary  design 
best  meeting  the  government  agency's 
needs  and  constraints  is  provided. 

The  specific  improvement  I have  in  mind  is  an 
increased  involvement  of  the  Government 
agency  in  the  two  processes  being  treated. 

This  paper  analyzes  the  work  process  involved 
in  systems  design  in  which  a set  of  user 
requirements  and  environmental  constraints 
are  converted  into  alternative  system  designs 
and  a preferred  design  selected.  It  identi- 
fies some  basic  problems  encountered  when  a 
systems  design  orgsnization  is  used  to 
initiate  this  process  of  designing  a new 
system  for  a client  organization.  These 
problems  fall  into  two  major  classes:  1)  how 

to  properly  state  the  requirements  and 
constraints  which  the  system  must  meet,  and 
2)  how  to  properly  evaluate  the  systems 
proposed  by  the  system  designers.  A major 
thesis  of  thie  paper  is  that  the  systems 
planning  process  is  a cooperative  effort 
betwean  tha  client  and  the  designer.  If  the 
latter  is  to  properly  design  a system  he  must 
not  only  thoroughly  understand  the  raquire- 
ments,  but  also  develop  an  evaluation  pro- 
cadure  which  is  acceptable  to  the  client  and 
meets  his  needs. 

An  analysis  of  various  evaluation  methods 
used  is  also  provided.  The  factors  often 
used  for  evaluation  include:  1)  System 

Performance  or  other  Technical  Factors,  2) 
Date  of  Availability  of  the  System,  3)  System 
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Cost,  4)  Risks  (in  performance,  schedule  and 
coat  which  arise  when  all  system  component* 
are  not  available  "off  the  shelf,"  but  some 
have  to  be  developed),  and  5)  other  miscel- 
laneous factors.  Generally  the  values  of 
each  hay  factor  for  various  alternatives  are 
assembled  in  matrix  form  for  validation  and 
comparison  purposes.  Unfortunately,  it  is 
rare  that  one  alternative  is  superior  to  all 
others  for  atl  descriptors  (often  the  al- 
ternative offering  superior  performance  is 
more  costly  or  has  higher  risk).  Thus  some 
means  of  relating  all  of  the  evaluation 
factors  must  be  used.  This  is  frequently 
done  by  applying  weighting  factors  (generally 
selected  hauristically),  which  causes  the 
final  "score"  to  be  highly  dependent  on  the 
values  of  the  weights.  Furthermore  clients 
often  have  difficulty  in  defending  this  ap- 
proach to  others  requiring  such  justification. 

This  paper  examines  in  detail  the  basic 

process  of  specifying  requirements,  creating 
des'ign  alternatives,  and  evaluating  them 
against  a set  of  criteria.  It  describes  a 
number  of  key  pitfalls  faced  by  the  systems 
designers  as  well  as  the  evaluators  which 
normally  occur  and  which  should  be  avoided 
during  a systems  planning  effort.  An  im- 
proved evaluation  process  avoiding  these 
pitfalls  is  presented  for  use  by  the  evalua- 
tion team,  allowing  them  to  select  the 

preferred  alternative  in  a more  rational, 
defensible  fashion.  Finally,  a method  of 
presenting  evidence  which  supports  and 
enhances  the  preferred  design  alternative  is 
described. 

While  the  suin  focus  of  the  paper  is  on  the 
architectural  design  process  in  which  there 
can  be  close  cooperation  between  the  system 
designers  and  the  client,  many  of  the  tech- 
niques described  also  apply  to  the  systems 

design  effort  which  occurs  during  a proposal 
generation  effort  when  such  cooperation  does 
not  exist.  Thus  the  paper  is  extended  to 

show  how  to  deal  with  these  problems  during  a 
proposal  effort. 

This  paper  builds  on  work  in  source  selection 
of  KDP  systems  previously  performed  by  the 
author  for  the  Assistant  Secretary  of  the  Air 
Force  (Financial  Management).  The  evaluation 
process  presented  now  includes  the  element  of 
developmental  uncertainties  which  was  not 
required  in  the  original  work.  While  the 
paper  has  greatest  value  for  contractors  and 
Government  agencies  involved  in  the  design  of 
large,  complex  systems  requiring  development, 
it  is  also  applicable  to  smaller  projects  in 
the  private  sector  as  well. 

PAST  I.  PROBLEM  DEFINITION 

PART  1 reviews  the  architectural  systems 
design  process,  indicating  the  various  work 
functions  involved,  and  the  information  re- 
quired by  a systems  designer  if  he  is  to  pro- 


vide a preferred  system  design  to  a client. 
Some  of  the  difficulties  in  obtaining  this 
information  are  described. 

1.  INTRODUCTION 

A msjor  objective  of  this  paper  is  to 
identify  a number  of  pitfalls  which  can 
prevent  a systems  designer  from  proposing  and 
designing  the  most  cost-effective  system  for 
a client,  taking  into  account  risks  and 
uncertainties,  and  to  indicate  ways  of 
avoiding  such  pitfalls. 

At  the  heart  of  these  problems  is  that  the 
entire  work  process  is  generally  divided 
among  those  major  contributors,  each  of  whom 
contributes  his  own  expertise  to  the  process: 

o The  system  user,  who  will  operate  and 
support  the  system  once  it  is  com- 
pleted, provides  a statement  of  his 
needs  (and  desires),  as  well  as  the 
environmental  constraints  which  are 
present. 

o The  system  designer  (generally  a con- 
tractor experienced  in  this  area)  who 
translates  the  user  needs  into  a set 
of  feasible  design  alternatives  making 
whatever  trade-offs  are  necessary,  and 
recoaaends  the  preferred  system. 

o The  procurement  organization  which 
serves  as  the  point  of  contact  with 
the  system  designer,  generally  is 
heavily  involved  in  the  evaluation  of 
the  design  alternatives  in  terms  of 
the  trade-offs  of  performance,  cost, 
availability  date  and  risk,  and  makes 
the  ultimate  decision  regarding  the 
preferred  system  selected. 


For  purposes  of  this  paper  we  shall  define 
these  participants  in  the  following  way: 

o The  term  "Client"  will  represent  the 
procurement  organization  who  is 
funding  the  architectural  study.  The 
client  will  be  responsible  for  obtain- 
ing the  specifications  from  the 
users.  Since  the  client  knows  the 
budgetary  constraints,  he  plays  a 
large  role  in  the  ultimate  systems 
evaluation  function  leading  to  the 
selection  of  the  preferred  system. 

o The  term  "Designer"  represents  the 
systems  analysis  and  design  organiza- 
tion who  has  been  contracted  to  per- 
form the  design  study. 


Although  a systems  designer  works  with  the 
client  under  varying  circumstances,  this 
paper  concentrates  on  two  disparate  situa- 
tions which  bound  most  of  the  set.  The  first 
example  is  an  architectural  design  effort  in 
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which  the  designer  maintains  a close  inter- 
face with  the  client.  In  the  secon  example, 
the  systems  designer  is  a vendor  proposing  a 
system  design  to  a client.  In  this  situation 
there  is  generally  minimal  contact  with  the 
client  during  preparation  of  the  design 
proposal. ^ 

2.  WORK  PROCESS  INVOLVED 

Both  situations  involve  a common  work 
process  which  typically  includes  these 
functions  (Figure  1): 

o Client  sets  system  specifications, 
including  desired  performance,  system 
availability  date,  and  constraints. 

o Designer  proposes  one  or  more  design 
alternatives  potentially  meeting  the 
system  specifications. 

o Designer  develops  an  evaluation  model 
based  on  his  understanding  of  the  job 
to  be  done  and  his  perception  of  the 
evaluation  model  to  be  used  by  the 
client. 

o Designer  uses  the  evaluation  model  to 
evalutle  alternative  system  design 
configurations  and  selects  the 
preferred  system  design. 

o Designer  submits  his  proposed  system 
design  to  the  client  for  his  evalu- 
ation. 

o Client  validates  that  all  specifica- 
tions have  been  met,  and  in  a design 
competition,  evaluates  all  proposals 
submitted  by  vendors,  and  selects  the 
preferred  proposal. 

o Client  makes  final  selection  of  the 
preferred  system. 


*This  situation  is  quite  common  in  the 
development  of  systems  for  the  government, 
particularly  the  Department  of  Defense. 


Figure  1.  Systems  Planning  Work  Process 


Having  presented  an  overview  of  this  process, 
we  shall  now  examine  each  of  the  steps  in 
greater  detail,  focusing  on  some  of  the 
pitfalls  which  may  arise. 

3.  POTENTIAL  PITFALLS  IN  THE  WORK  PROCESS 

It  should  be  obvious  that  the  two  major 
drivers  of  the  design  effort  are  the  de- 
signer's understanding  and  perception  of:  1) 
the  client's  specifications,  and  2)  the 

client's  evaluation  model.  Unless  the 

designer  understands  what  the  client  has 
specified,  the  final  system  design  may  be 
configured  to  produce  the  wrong  system. 
Specifically,  unless  the  designer  knows  and 
understands  the  evaluation  model  to  be  used 
by  the  client,  the  designer  will  not  be  able 
to  properly  make  his  performance,  cost, 
availability,  and  risk  trade-offs  to  produce 
the  "optimal"  system  desired.  During  an 

architectural  design  effort,  there  are 

usually  opportunities  to  meet  with  the  client 
to  obtain  a good  mutual  understanding  of  what 
the  client  really  desires  (the  "real"  system 
specifications),  as  well  as  the  proper  evalu- 
ation model  which  should  be  used.  Unfortu- 
nately, this  type  of  information  is  generally 
not  available  from  the  client  during  a 
proposal  effort.  Thus  in  the  next  four 
sections  we  shall  describe  a process  for 
providing  designers  with  a better  understand- 
ing of  system  requirements  and  the  system 
evaluation  process,  using  the  case  of  an 
architectural  study  as  an  example.  We  shall 
then  consider  the  analogous  planning  problem 
which  should  occur  during  a proposal  effort. 

4.  UNDERSTANDING  THE  SYSTEM  SPECIFICATIONS 

The  first  part  of  a client's  specifica- 
tions typically  describes  characteristics 
needed  by  the  designer  to  synthesize  the 
system.  Sometimes  these  descriptors  indicate 
the  concept  of  operation,  the  missions, 
functions  and  jobs  to  be  done,  and  the 
performance  characteristics  (e.g.,  speed  of 
response,  reliability  and  maintainability) 
required.  Sometimes  these  descriptors 
consist  of  a set  of  technical  or  design 
characteristics  (e.g.,  core  size  and  number 
and  type  of  displays  for  a data  processing 
system). 

The  second  part  of  the  specifications  may 
consist  of  a set  of  environmental  or  opera- 
tional constraints  that  must  be  observed. 
This  set  could  include  operational  tempera- 
ture, humidity,  shock  and  vibration,  as  well 
as  specifications  which  must  be  met  so  that 
this  system  can  interface  with  other  systems. 
In  addition,  the  set  of  requirements  should 
include  the  date  when  the  system  must  be 
operational. 

These  specifications  are  generally  stated  in 
two  ways.  The  first  is  a set  of  minimum 
mandatory  requirements  that  must  be  met  or 
else  the  design  will  be  considered  non- 
responsive  to  the  specifications.  Sometimes 
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the  client  indicate*  a detire  for  additional 
capabilitie*  if  they  are  available.  These 
"desirable  features"  are  not  aade  part  of  the 
aandatory  requirements  since  the  designer  may 
not  be  capable  of  providing  these. 

Early  in  the  planning  effort,  the  designer 
should  review  the  system  specifications.  Any 
questions  about  these  should  be  resolved 
during  a conference  with  the  client  early  in 
the  design  study  effort. 

5.  UHDgRSTAHDIHG  THIS  EVALUATION  MODEL 

Having  established 'a  mutual  understanding 
of  the  initial  system  "requirements"  as  the 
baseline  for  the  architectural  design  study, 
the  next  step  is  to  obtain  a mutual  agreement 
with  the  client  of  the  evaluation  model  to  be 
used.  Here  we  are  concerned  with  three  major 
points: 

o The  evaluation  model  or  method  should 
be  explicit  so  that  the  designer  can 
perform  various  cost-performance 
trade-offs  to  arrive  at  a preferred 
solution. 

o The  evaluation  model  should  be  ra- 
tional, credible  and  defensible. 

o The  evaluation  model  should  be  agreed 
to  by  the  client.  If  not,  the  final 
results  obtained  may  not  be  accept- 
able. 

With  the  preceding  discussion  in  mind,  let  us 
now  examine  various  evaluation  methods  which 
are  used  by  various  government  agencies,  and 
describe  how  designers  may  respond  to 
each.1  This  will  be  helpful  in  determining 
whether  such  responses  are  desirable,  or  if 
the  evaluation  method  should  be  modified 
accordingly  to  produce  the  results  desired. 

5.1  The  "Extra  Performance  Is  Overkill" 

Evaluation  Method 

The  first  evaluation  method  examined, 
illustrated  in  Figure  2,  operates  as  follows: 

o All  evaluation  factors  and  their 
minimum  mandatory  requirements  as 
contained  in  the  Statement  of  Work 
(SOW)  are  listed  in  coluam  form. 

o The  actual  values  provided  by  each 
alternative  system  being  evaluated  are 
then  listed  and  validated  by  the 
evaluator  that  these  values  each  meet 
its  requirement. 

o Any  system  characteristic  which  is 
over  the  minimum  level  specified  can 


^While  this  discussion  specifically  applies 
to  government  contracts,  it  also  applies  to 
many  non-government  contracts  as  well. 


be  considered  "overkill",  and  has  no 
additional  value  as  compared  to  the 
minimum  level. 

o The  selection  criterion  used  is  as 
follows:  choose  that  system  whose 

characteristics  individually  meet.  or 
exceed  all  constraints  and  minimum 
specifications,  and  whose  Present 
Value  Life  Cycle  Cost  (PVLCC)  is  least 
among  all  alternatives  under  con- 
sideration. 

The  main  advantage  of  this  approach  is  that 
it  explicitly  states  the  "rules  of  the 
game".  Ideally  each  system  would  be  designed 
to  exactly  equal  the  design  specifications 
since  any  larger  values  would  generally 
result  in  higher  cost.  In  this  case  the 
evaluator  might  select  as  the  preferred 
system  alternative  the  one  which  has  the 
least  PVLCC. 

Unfortunately  system  components  come  in  dis- 
crete units  rather  than  from  a continuous 
array,  and  the  specifications  of  similar 
units  generally  differ  from  vendor  to 
vendor.  Thus  to  be  entirely  responsive  to 

the  specifications  which  have  been  issued, 
the  only  feasible  design  solution  may  be  to 
use  components  which  individually  meet  or 
surpass  the  minimum  requiresients  which  have 
been  stated.  In  this  case,  excess  perform- 
ance may  be  provided,  and  the  key  deficiency 
of  this  approach  is  that  extra  performance  is 
ignored.  It  should  not  be.  There  may  be 

justification  in  giving  some  extra  credit  for 


EVALUATION  MATRIX 


Validate  That  Each  Vendor  Meets  All  Minimum  Requirements 
Excess  Performance  Is  “Overkill" 

Choose  by  PVLCC 

Pitfalls: 

e Excess  Performance  Has  Worth 
e Trade-Otfs  Among  Factors  Are  Possible 


Figure  2.  Evaluation  Method  1 
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extra  performance  to  counter  balance  the 
additional  cost  which  generally  accompanies 
extra  performance. 

If  extra  credit  cannot  be  given  for  excess 
performance,  the  designer  might  be  permitted 
to  compensate  for  a performance  deficiency  by 
providing  excess  value  of  some  related  char- 
acteristic. For  example,  the  same  probabil- 
ity of  kill  for  a missile  could  be  obtained 
by  either  having  a highly  accurate  guidance 
system  and  a low  yield  warhead,  or  having  a 
lower  accuracy  guidance  system  and  a higher 
yi#Ld  lUrKrmfi-  Thtm  it  mj»f  hp  possible  to 
achieve  the  same  end  result  at  lower  cost  to 
the  client  by  using  a component  which  may  not 
quite  meet  an  "arbitrary"  minimum  specifica- 
tion if  another  related,  higher  value  com- 
ponent is  used  as  compensation.  The  designer 
can  best  make  such  determinations  since  the 
designer  usually  knows  the  relationship 
between  performance  characteristics  and  cost, 
Sod  Wtc  re,  wzldr  whUk  &st  uf  hie  a li- 
able characteristics  can  best  perform  a 
defined  job  (within  constraints)  at  lowest 
total  cost. 

Thus  the  major  improvements  which  can  be  made 
to  the  "Extra  Performance  is  Overkill”  method 
are  to  define  the  jobs  to  be  done  at  the 
mission  or  functional  level  and  allow  inter- 
system trade-offs  to  be  made  within  a con- 
strained set  of  boundary  conditions.  Note 
that  the  system  design  task  will  also  permit 
trade-offs  between  quality  and  quantity. 
Thus,  in  the  case  of  a missile  system,  if  the 
probability  of  kill  of  one  missile  is  greater 
than  another,  it  may  be  possible  to  configure 
both  systems  to  achieve  a given  level  of  tar- 
get destruction,  in  which  a lower  performance 
missile  will  require  the  use  of  more  missiles 
to  do  the  same  job  than  a higher  performance 
missile.  The  PVLCC  calculations  will  deter- 
mine the  preferred  system.  To  protect  the 
client  against  unacceptable  design  features 
(such  as  tbe  proposal  of  a very  low  perform- 
ance missile  in  the  previous  example),  the 
client  can  specify  a minimum  value  that  must 
be  provided  for  any  characteristic  (such  as 
the  missile  probability  of  kill  must  exceed 
0.5). 


5.2  The  "Point  Scoring"  Evaluation  Method 

Another  disadvantage  of  the  previous 
evaluation  method  is  as  follows.  The  client 
may  have  in  mind  a minimum  level  of  capa- 
bility, but  may  desire  additional  capability 
if  obtainable  at  a reasonable  cost.  Thus, 
some  way  must  be  found  to  give  "additional 
credit"  to  those  vendors  which  can  provide 
these  desirable  features  or  superior  char- 
acteristics beyond  the  minimum  specifica- 
tions. This  can  be  accomplished  by  using  the 
so-called  "point  scoring"  method,  illustrated 
in  Figure  3.  In  this  method  the  key  evalua- 
tion factors  are  listed  again  as  one  dimen- 
sion of  the  evaluation  matrix,  and  their 
values  for  each  alternative  constitute  the 


other  dimension.  As  in  the  previous  evalua- 
tion method,  the  next  step  in  this  evaluation 
method  is  to  validate  that  each  of  the 
mandatory  requirements  has  been  met.  Then 
each  of  the  key  factors  where  a value  other 
than  a fixed  mandatory  value  is  desired  is 
assigned  two  numbers  which  will  translate  its 
value  into  a point  score.  The  first  number 
(V  in  Figure  3)  translates  the  extra  amount 
of  performance  provided  into  a normalized 
value  (say  from  0 to  10  to  normalize  the 
worth  of  each  factor).  The  second  number  (W 
in  Figure  3)  provides  the  Weighting  Factor  or 
j«*lativh  vjrl'i'  r*f  rhi«  fsi'r  jr  t-i.1  a1! 

of  the  other  factors  involved.  For  example, 
cost  may  constitute  60Z  of  the  total  score 
possible.  Choosing  the  latter  as  1000 
points,  the  value  of  W for  cost  may  be  chosen 
as  600  points.  Thus  the  lowest  cost  design 
could  be  given  a V • 1,  and  each  design  would 
receive  a V equal  to  the  ratio  of  the  cost  of 
the  lowest  cost  design  to  the  cost  of  the 
design  'uJ«:  twiluiUv.  th»-  if  tort 
were  twice  as  much  as  another  alternative, 
the  lowest  cost  system  would  receive  600 
points  and  the  other  system  would  receive  300 
points.  These  numerical  values  chosen  for  V 
and  W would  be  based  either  on  available 
operational  data  or  on  the  judgment  of  the 
technical  evaluators. 

Each  system  alternative  would  then  be  evalu- 
ated with  respect  to  each  factor  in  order  to 
determine  how  many  of  the  maximum'  points 
allocated  would  go  to  each  of  the  proposed 
alternatives.  A total  score  for  each  al- 
ternative is  then  obtained  by  sunning  each  of 
its  factor  scores. 

This  method  does  have  the  advantage  of 
providing  credits  for  extra  performance. 


EVALUATION  METHOD  2 
POINT  SCORING  METHOD 
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However,  it  also  has  several  difficulties. 
First,  while  the  key  factors  contributing  to 
the  worth  of  a system  way  be  identified,  the 
use  of  value  and  freighting  factors  (V  and  W) 
as  the  method  of  combining  factors  is  always 
subject  to  challenge  by  other  evaluators  or 
decisionmakers.  Thus,  what  is  needed  is  a 
more  defensible  way  of  combining  the  factors 
listed. 


among  the  system  alternatives.  Thus,  while 
these  "siatrix  evaluation  methods"  enable  the 
evaluator  to  rapidly  focus  on  the  relative 
differences  among  systems,  they  have  basic 
flaws  as  positive  selectors  of  the  preferred 
system. 

PAST  II.  GENERATING  AN  IMPROVED  EVALUATION 
METHOD 


The  second  difficulty  inherent  in  the  point- 
V-  scoring  method  is  even  more  serious.  This 

'.-*j  method  combines  cost  values  with  the  tech- 

l»;3  nical  or  performance  values  through  the 

vehicle  of  points.  Yet  while  selecting  the 
preferred  system  based  on  highest  score  is 
intuitively  sound,  there  is  no  scientific 
justification  for  the  use  of  such  a "figure 
of  merit"  approach.  There  are  two  more 

widely  accepted  methods  of  selecting  a 
preferred  alternative.  The  first  is  to 

select  that  system  alternative  which  will 
perform  the  operational  functions  and  meet 
all  constraints  at  the  lowest  total  cost  to  a 
defined  organisation  (i.e.,  pivoting  on  equal 
effectiveness).  The  second  approach  is  to 
select  that  alternative  which  will  yield  the 
highest  performance  of  the  operational 
functions  at  a fixed  total  cost  (i.e., 
pivoting  on  equal  cost).  Such  a method  must 
also  take  into  account  the  risks  and  uncer- 
tainties involved. 

Lastly,  experience  has  shown  that  when  a 
large  list  of  factors  are  included  in  the 
evaluation,  the  final  score  for  each  system 
is  often  very  close  to  one  another,  rendering 
this  evaluation  method  ineffective.  One 
reason  for  this  closeness  in  score  is  because 
the  value  of  most  of  the  large  number  of 
factors  being  added  together  are  fairly  close 
to  one  another  since  most  values  correspond 
to  the  minimum  mandatory  requirements.  These 
values  overpower  value  of  the  few  remaining 
factors  which  describe  the  real  differences 


PAST  II  returns  tv  v;n (laments Is  and  analyses 
the  key  factors  which  represent  the  results 
of  an  effort  of  developing  and  constructing  a 
system  which  involves  components  that  are 
either  beyond  the  state  of  the  art  or  are  not 
readily  available  "off-the-shelf"  and  thus 
have  to  be  developed.  From  this  scenario  we 
develop  an  improved  method  for  evaluating 
proposed  system  alternatives. 

6.  CONSIDERATION  OF  DATE  OF  AVAILABILITY 
AND  ITS  UNCERTAINTY 

The  previous  discussion  of  the  two 
commonly  used  evaluation  methods  and  their 
deficiencies  concentrated  on  the  two  key 
evaluation  factors  of  system  performance 
(getting  the  jobs  done)  and  cost.  In  this 
section  we  shall  consider  two  other  evalua- 
tion factors  which  oust  be  considered  when 
the  design  contains  elements  which  must  be 
developed.  In  this  case  the  system  designer 
(and  client)  must  also  consider  technological 
or  developmental  uncertainties  which  are 
further  reflected  into: 

o The  date  when  the  system  will  be 
available  for  use 

o The  final  system  cost 

o The  system  performance  achieved 

To  perform  this  analysis  we  need  to  consider 
the  entire  effort  of  developing  and  con- 
structing the  system  as  a work  process  which 
can  be  modeled  as  a series-parallel  network 
as  shown  in  Figure  4.  This  network  indicates 


Include  All  Work  Elements:  RDT&E,  Procurement,  Installation,  Operations,  Maintenance,  Support 

Indicate  Required  Deliverables 

Indicate  Completion  Time  and  Their  Uncertainties 


Figure  4.  Project  Activities  Network 
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that  this  entire  work  effort  (defined  as  "the 
project")  consists  of  a group  of  work  activi- 
ties arranged  in  a preferred  sequence  or 
order.  Some  of  these  activities  (when 
completed  satisfactorily)  produce  outputs  or 
deliverables  required  as  part  of  the  State- 
ment of  Work.  Each  of  the  activities  re- 
quires time  and  the  expenditure  of  manpower 
and  other  resources.  Thus  using  Critical 
Path  Scheduling  techniques  the  network  can  be 
analysed  and  the  project  completion  time  can 
be  calculated  (based  on  the  sum  of  the  times 
of  those  activities  along  the  "critical 
path").  In  addition,  the  man-hours  required 
can  be  sumaed  and  converted  into  manpower 
costs  and  total  costs. 

Having  described  the  project  effort  as  a work 
process,  two  observations  can  be  made. 
First,  the  entire  project  effort  can  be 
completed  in  an  acceptable  fashion  only  if 
all  of  the  various  work  activities  shown  in 
figure  4 are  completed  in  the  sequence  shown, 
resulting  in  the  completion  of  the  various 
required  deliverables.  It  can  be  assumed 
that  if  an  activity  is  not  completed  satis- 
factorily, the  project  effort  cannot  continue 
and  it  will  be  aborted,  unless  a complemen- 
tary activity  which  should  also  be  shown  in 
the  network  can  be  completed  as  a substitute. 

Secondly,  given  that  an  activity  is  completed 
satisfactorily,  the  time  and  man-hours 
required  for  such  completion  can  rarely  be 
estimated  exactly  for  all  development  type 
activities.  This  uncertainty  in  completion 
time  can  best  be  represented  by  a three-point 
estimate!  the  moat  likely  value,  and  the 
limits  of  uncertainty  at  the  5th  to  the  95th 
percentile,  as  illustrated  in  figure  5a.  The 


completion  time  of  the  entire  project  will 
similarly  have  a range  of  uncertainty  as 
illustrated  by  the  probability  distribution 
of  figures  5b  and  5c. 

6.1  Evaluation  Calculations  to  be  Made 

To  aimplify  the  calculations  involved, 
}t  will  be  assumed  that  the  project  activity 
network  constructed  describes  exactly  the 
work  process  to  be  employed.  Since  this 
assumption  will  be  applied  to  all  slterna- 
tives,  the  relative  accuracy  of  the  evalua- 
tion she u Id  not  be  greatly  impaired.  Here 
are  the  calculations  to  be  made: 

o Determine  the  level  of  acceptability 
for  each  activity  in  the  network. 
Assign  the  planned  resources  to  each 
activity  and  provide  a three-point 
estimate  of  the  time  required  to 
complete  each  activity  in  an  accept- 
able fashion  using  these  resources. 

o Estimate  the  maximum  time  each 

activity  will  be  permitted  to  con- 
tinue before  the  activity,  and  hence 
the  project,  will  be  terminated.  If 
desired,  parallel  paths  can  be 

inserted  into  the  network  to  reduce 
the  chsnce  of  project  termination. 

o From  the  individual  probabilities  of 
activity  failure,  calculate  the 

probability  of  project  termination 
(failure).  Then  calculate  the  proj- 
ect completion  time  as  a probability 
distribution  when  the  project  is  suc- 
cessful, as  illustrated  in  Figure  5c. 

o Using  the  same  time  estimates, 
calculate  the  manpower  cost  and  other 
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coat*  associated  with  each  activity 
and  the  entire  project  aa  a probabil- 
ity distribution.  This  will  be  simi- 
lar to  Figure  Sb. 


project  in  leas  than  the  maximum  acceptable 
date.  The  resulting  capabilities,  dates,  and 
total  costs  are  as  shown  in  the  three- 
dimensional,  bell-shaped  set  of  points. 


6.2  Representing  the  Project  Results 

From  the  previous  calculations,  the  key 
evaluation  characteristics  of  the  project  may 
be  expressed  as  a three-dimensional  probabil- 
ity distribution,  as  illustrated  in  figure 
6.  This  figure  should  be  interpreted  as 
illustrating  the  statistical  set  of  results 
of  performing  all  development  activities  a 
large  nunber  of  times.  Each  vector  (having 
one  of  the  end  points  shown)  represents  one 
of  the  results  described  by  each  of  thoae 
coordinates:  1)  the  system  capability  of 

meeting  the  entire  set  of  mandatory  specifi- 
cations, 2)  the  date  when  the  system  will 
become  available,  and  3)  the  total  cost 
required  to  obtain  the  results.  Applied  to 
these  results  are  two  threshold  levels  of 
acceptability  of:  1)  minimum  level  of  system 

capability,  and  2)  maximum  allowable  avail- 
ability date.  Applying  these  levels  of 

acceptability  (to  both  the  project  deliver- 
ables and  to  the  system  implementation 
process  itself  as  it  progresses),  from  a 
statistical  point  of  view,  it  can  be  seen 
that  certain  of  these  "trials"  are  defined  as 
baing  "unsuccessful!',  since  they  do  not  meet 
the  minimum  lavel  of  acceptability.  These 
trials  result  in  sero  system  capability,  but 
do  consume  both  time  and  cost,  as  represented 
by  tha  cluster  of  points  on  the  YZ  plan  (sero 
system  capability).  Note  that  the  times 
spent  on  the  project  vary  from  early  cancel- 
lation of  tha  projact  to  later  cancellation. 
Costs  of  the  unsuccessful  project  "trials" 
are  also  shown.  The  other  points  of  Figure  6 
raprasent  the  rasults  of  the  successful 
"trials."  Note  that  all  successful  trials 
meet  at  least  the  minimum  level  of  system 
capability  and  all  trials  complete  the 


This  method  of  analysing  and  evaluating 
results  shows  that  performance  risk  can  be 
defined  as  the  probability  of  meeting  the 
minimum  set  of  requirements.  By  making  the 
assumption  that  all  activities  (of  the 
project  network)  are  independent  of  one 

another  and  each  must  be  'completed  by  some 
specified  date  (or  the  entire  project  will  be 
terminated),  the  probability  of  project 
success  can  be  calculated  as  the  joint 
probability  that  all  activities  will  be 
successful  (the  product  of  the  probabilities 
of  success  of  all  activities). 1 

Schedule  Risk  and  Cost  Risk  will  be 
treated  in  a later  section. 

7.  DEFINING  THE  BVALPATION  OBJECTIVE 

Having  defined  the  analytical  structure 
for  the  evaluation  approach,  we  can  now 
explicitly  define  the  selection  objective  as 
follows: 

"To  select  a proposed  system  which  performs  a 
set  of  future  required  jobs  at  given  work 
load  levels  and  meets  all  required  con- 
straints including  maximum  availability  date 
at  the  lowest  total  cost,  taking  into  account 
all  uncertainties. " 

This  objective  includes  the  following  three 
major  concepts: 


1-Note  that  if  the  assumption  of  activity 
independence  is  not  acceptable,  a similar, 
but  more  difficult,  analysis  can  be  side 
taking  the  pertinent  dependencies  into 
account. 
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a.  Tha  contractor  muat  «how  that  each 
of  his  system  alternatives  can  perform  all  of 
tha  future  jobs  and  meat  all  the  constraints. 

b.  Lowest  total  present  valua  life 
cycla  cost  should  be  the  selection  criterion. 

c.  Development  and  job  uncartaintiaa 
ara  tha  kay  factors  which  maka  tha  evaluation 
salaction  procasa  a difficult  ona. 

8.  SUMMARY  OF  SYSTEMS  PLANNING  APPROACH 

Figure  7 summarises  tha  atepa  to  be 
followed  in  an  architectural  design  study: 

a.  The  cliant  will  define  ona  or  more 
sata  of  mandatory  system  requirements  in 
tanas  of  jobs  to  ba  done  and  conatraints  to 
be  met.  The  client  will  alao  specify  a set 
of  dasirabla  faaturas  over  and  above  the 
mandatory  requirements  which  thay  would  like 
to  obtain,  if  possible,  and  a sat  of  jobs 
which  would  usa  such  desirable  featuras. 

b.  Tha  designar  will  dasign  a number  of 
syatem  alternatives  which  meet  each  aat  of 
mandatory  requirements,  on  or  bafora  a speci- 
fied availability  data,  at  a leval  of  risk 
specified  by  tha  cliant. 

c.  The  deaignar  will  calculate  tha 
presant  value  life  cycle  cost  (PVLCC)  of 
Mating  the  statad  set  of  requirements  at  a 
level  of  risk  specified  by  the  client. 

d.  The  designer  will  also  provide  total 
cost  data  relevant  to  providing  and  operating 
each  desirabla  feature  he  provides,  as  em- 
bedded in  the  reprasentative  joba  for  which 
the  desirable  feature  is  to  be  used. 

a.  Based  on  this  data,  tha  designer 
will  calculate  the  cost  of  performing  the  set 


of  jobs  aasociatad  with  aach  Dasirabla  Fea- 
ture proposed  by  tha  designer.  This  cost 
will  be  compared  against  the  cost  of  perform- 
ing the  sat  of  joba  if  tha  proposad  Desirable 
Feature  were  not  available.  For  aach  of 
thase  seta  of  jcba,  the  least  costly  way  of 
performing  ttheae  joba  will  be  choaan  and  this 
cost  added  to  the  cost  of  performing  tha 
mandatory  jobs. 

f.  Theae  results  (tha  preferred  ayatam 
for  aach  lavel  of  aystem  capability)  will  ba 
shown  to  the  cliant. 

g.  The  client  will  salact  the  final 
preferred  aystem  based  on  a comparison  of  the 
incremental  cost  to  tha  incremental  gain  for 
increasing  lavels  of  system  capability. 

PART  III.  APPLYING  EVALUATION  METHOD  IN  AN 
ARCHITECTURAL  DESIGN  STUDY 

PART  III  amplifiaa  tha  description  of  tha 
design  approach  by  showing  how  to  apply  the 
approach  in  an  architactural  dasign  study. 

9.  AN  EXAM  LB  OF  THIS  PROPOSED  APPROACH 

Having  de sc rib ad  tha  approach  to  be 
followed,  we  shall  now  considar  an  example  of 
how  tha  approach  would  operate  in  practice. 
The  example  usad  is  that  of  an  architectural 
dasign  study  of  an  information  system. 

9.1  Cliant  Issues  the  Total  Set  of  Require- 
ments 

As  mentioned  previously,  this  would 
include: 

o The  basic  system  workload  (as  char- 
acterised by  a reprasentative  sat  of 
EDP  jobs),  in  terms  of  the  mandatory 


1.  Client  will  specify  alternative  sets  of  design  requirements  in 
terms  of  jobs  to  be  done,  performance  and  constraints. 

2.  Client  will  specify  minimum  mandatory  requirements  and 
extra  features  desired. 

3.  Constrain  designers  to  provide  alternative  designs  which 
meet  the  set  of  system  requirements,  including  maximum 
availability  date  and  level  of  risk. 

4.  For  each  alternative,  calculate  the  present  value  life-cycle 
cost  (PVLCC). 

5 Determine  the  worth  of  any  significant  extra  performance. 

6 Select  the  system  which  meets  the  set  of  requirements 
and  constraints  at  lowest  PVLCC  to  client  while  accounting 
for  risks  and  uncertainties 

7.  Show  preliminary  results  to  client  and  trade-off  specifications 
with  feasible  system  results 

8 Designer  assists  the  client  in  finalizing  on  Preferred  System 


Figure  7.  Summary  or  Architactural  Dasign  Approach 
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capabilities  to  be  set  over  time,  as 
illustrated  in  Figure  8 

o All  mandatory  constraints  to  be  met 

o A statement  of  Desirable  Features  (or 
extra  capability  desired) , and  a 
statement  of  the  set  of  jobs  for 
which  each  desirable  feature  would  be 
used  if  provided  by  the  designer 

9.2  Consideration  of  Mandatory  System 

Capability 

The  first  step  which  the  designer  must 
take  is  to  configure  one  or  more  system 
alternatives  which  will  meet  or  exceed  the 
minimum  mandatory  workload  requirement. 
Figure  8a  shows  the  "input  demand  function" 
(in  this  case  a workload  which  will  be 
increasing  over  time).  The  increase  shown  is 
expected  to  be  gradual  from  start  until  tj, 
when  a large  increase  is  expected.  The 
workload  then  continues  to  increase  gradually 
until  t2  when  a second  large  increase  will 
occur.  After  t2  the  increase  is  again 
gradual.  The  planned  system  life  is  five 
years,  which  occurs  at  t3> 

In  the  example  being  presented,  the  objective 
of  the  system  is  to  provide  sufficient  capa- 
bility to  process  the  forecasted  daily 
workload  within  a 24-hour  period.  However, 
as  shown  in  Figure  8b,  the  24-hour  period 
must  also  include  time  for  Rework  to  correct 
all  errors  detected,  and  Down  Time  (for  both 
preventive  and  corrective  maintenance).  With 
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this  in  mind  we  shall  define  the  term  "Reli- 
ability" to  represent  the  frequency  of 
out  1 functions,  and  "Maintainability”  to  repre- 
sent the  amount  of  time  when  the  system  has 
reduced  capability  due  to  required  repairs  or 
replacement.  Thus  the  system  designer  must 
make  certain  that  the  uet  system  capability 
will  enable  the  workload  to  be  processed  in 
the  required  time. 

These  failure  and  down  time  considerations 
are  illustrated  in  Figure  9.  System  Avail- 
ability is  defined  as  the  proportion  of  Up 
Time  to  Total  Time.  The  set  of  these  factors 
may  be  treated  in  the  following  way: 

a.  Make  certain  that  the  total  down 
time  and  associated  reduction  in  system 
capability  is  taken  into  account  when  design- 
ing the  system  to  meet  the  required  workload. 

b.  Frequency  of  failure  or  system 
availability  may  also  be  treated  as  a system 
constraint,  i.e.,  the  maximum  frequency  of 
failure  that  can  be  tolerated. 

c.  All  of  the  maintenance  factors 
finally  result  in  added  cost,  and  will  be 
accounted  for  in  the  Present  Value  Life  Cycle 
Cost  (PVLCC)  analysis  of  each  system. 

9.3  Dealing  With  an  Uncertain  Workload 

The  previous  analysis  of  system  capa- 
bility was  based  on  the  assumption  that  the 
input  workload  was  known  exactly,  as  illus- 
trated in  Figure  8a.  Sometimes  the  assump- 
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tion  is  made  thet  this  is  the  minimum  manda- 
tory workload  but  that  extra  credit  will  be 
given  for  systems  having  the  capability 
greater  than  the  minimum.  The  designer's 
problem  is,  how  much  extra  capability  is 
desired  over  time?  Many  times  the  client 
cennot  accurately  predict  what  the  actual 
workload  may  be.  However,  some  limits  must 
be  set  if  appropriate  guidance  is  to  be  given 
to  the  designers. 

One  way  of  dealing  with  this  uncerteinty  is 
to  express  the  workload  as  a probebility 
distribution  as  shown  in  Figure  10.  Setting 
the  upper  limit  is  fairly  straightforwerd, 
since  this  can  be  set  as  an  arbitrery  design 
limit  beyond  which  additional  capability  is 
assumed  to  have  no  velue.  Intermediate 
probability  values  can  then  be  inserted,  as 
shown  in  Figure  10,  using  whatever  dete  the 
client  has  evailable  (either  statistical  deta 
or  judgmental  estimates). 

Based  on  this  assumed  workloed,  the  designer 
must  then  design  the  system  to  be  able  to 
meet  the  entire  range  of  workload  levels, 
over  time,  adding  edditional  system  incre- 
ments whenever  required.  In  the  illustration 
of  Figure  10,  the  designer  proposed  System 
A}  as  the  initial  system.  This  system  will 
"absolutely"  meet  the  workload  requirement  in 
Years  1 and  2 and  will  "absolutely  not"  meet 
the  requirement  in  Yeer  5.  However,  the 
designer  proposed  to  add  an  additional 

capability  to  A},  to  yield  System  A2, 
whenever  Aj  is  needed.  Generally  two 

constraints  are  pieced  on  the  designer  for 
purposes  of  design  and  evaluation! 
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o No  more  than  one  or  two  growth  addi- 
tions will  be  permitted  to  keep 
disruptions  within  acceptable  limits. 

o An  addition  will  be  required  whenever 
the  operational  hours  per  month 
reaches  some  upper  limit  (say,  600 
hours  per  month  as  shown  in  Figure 
10).  This  will  permit  sufficient 
time  for  corrective  and  preventive 
maintenance.  Alternatively,  this 
limit  could  be  a function  of  the 
system's  demonstrated  maintainability. 

Providing  such  system  "upgrades"  during  its 
operational  life  saves  the  user  money  since 
it  prevents  the  user  from  having  to  buy  more 
capability  than  is  required  (like  A2)  at 
the  beginning  of  system  operations  rather 
than  when  it  is  actually  needed. 

Finally  the  designer  should  provide  "credible 
evidence"  for  the  client  that  the  preferred 
system  can  in  fact  meet  the  entire  defined 
workload  (es  shown  in  Figure  10)  over  its 
entire  levels,  as  well  as  all  constraints. 
For  off-the-shelf  systems,  this  could  be 
validated  by  live  test  demonstrations.  For 
development  type  projects  this  could  be  shown 
by  simuletion  or  enalysis. 

9.4  Aveilabilitv  Petes  and  Schedule  Bisk 

Given  that  each  system  hes  been  designed 
to  provide  the  required  net  productivity  to 
meet  the  incoming  workload  specified  in 
Figures  8a  or  10,  we  shall  now  describe  how 
the  designer  plans  the  development  effort  to 
meet  the  required  availability  detes  ( tj 
and  tj)  end  provides  the  necessery  dete  to 
the  client,  enabling  him  to  perform  his 
validation  function.  Recall  that  Figure  4, 
which  displays  the  project  work  process  in 
network  form  and  the  time  estimates  of  the 
activities  elong  the  critical  path,  was  used 
to  calculate  the  project  completion  date  as 
the  normal  distribution  of  Figure  Sc.  In  the 
example  shown,  there  is  approximately  a 40Z 
chance  of  successfully  completing  the  project 
by  date  Tj  (60Z  chance  of  schedule  over- 
run). However,  the  client  may  not  be 

comfortable  with  this  degree  of  risk.  Thus, 
the  system  designer  must  find  out  whet  risk 
of  overrun  the  client  is  willing  to  accept. 
Assume  e 10Z  risk  is  acceptable.  As  shown  in 
Figure  Sc,  for  this  10Z  risk  the  project 
would  be  completed  on  or  before  T2  which  is 
later  than  T}  and  hence  unacceptable. 

Thus,  the  syrtem  designer  must  reconfigure 
the  project  plan  to  reduce  completion  dete. 
Generally  this  is  done  by  adding  more  re- 
sources on  one  or  more  activities  on  the 
critical  path  until  the  new  completion  dete 
probability  function  satisfying  the  require- 
ment (10Z  chance  of  overrun  at  time  Tj)  is 
obtained,  as  shown  in  Figure  Sc. 

In  summary,  Schedule  Risk  is  defined  as  the 
probability  that  the  project  will  overrun  a 
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required  delivery  dete.  Fcr  purpose*  of  the 
planning  end  evaluation  efforts,  the  value  of 
Schedule  Risk  acceptable  to  the  client  should 
be  provided  to  the  designer  at  tha  beginning 
of  the  study*  Than  it  is  up  to  tha  designer 
to  construct  the  project  work  plan  accord- 
ingly end  provide  tha  projact  activities 
nstwork  plus  *11  tine  calculations  to  show 
that  the  coaplation  data,  as  a probability 
function,  satisfies  both  the  tine  and  risk 
requirements. 


9*5  Consideration  of  Total  Cost 

The  project  activities  network  (Fig- 
ure 4)  is  also  tha  starting  point  for  tha 
cost  calculations.  Fron  this  network  tha 


cost  of  *11  work  and  cost  elements  (RDT&E, 
procurement,  installation,  operations, 
maintenance,  end  support)  borne  by  tha  cliant 
**ust  be  calculated  for  each  year.1  Hera 

lilt  •tf*"<,*r<1  f°rTul“  *or  calculating  th. 
cost  of  an  activity  ere  used  (e.g.,  labor 

cost  equals  the  product  of  man-hours  and 
labor  rate).  when  the  activity  tIL  it 
expressed  es  * three-point  estimate,  thesa 

llTl  “ exP*cted  valu. 

in  FiwA  slI  Vi*ti0“  (m  ,h0TO  Previously 
in  Figure  5e).  These  values  for  time  ara 

than  converted  to  l.bor  cost  value,  by 
■.Implying  each  by  tha  labor  rate.  Cost  I 

ere  then  accuwiiated  by  year,  and  tha 
expected  value  of  cost  and  the  cost  variance 
for  each  year  are  calculated  aa  follows:  The 

•*pacfad  total  cost  for  the  year  is  equal  to 
the  sum  of  the  expected  costa.  The  total 
cost  variance  for  each  year  is  equal  to  the 
sum  of  the  squares  of  tha  individual  standard 
deviation*  for  that  year. 


Hhen  * probabilistic  workload  is  assumed  (as 
discussed  in  the  previous  section),  each 
yearly  cost  must  ha  handled  as  a frequency 
distribution.  Operations,  maintenance,  or 
leasing  costs  must  be  calculated  for  each 
probability  segment.  Using  Figura  10  as  an 
example,  first  celculeta  tha  cost  of  sagnsnt 
*1  • function  of  its  operating  time. 

Using  the  time  of  the  mid-point  of  the  p} 
segment,  end  using  the  pricing  date  supplied 
the  designer  as  well  as  tha  user  labor  cost, 
calculate  the  cost  associated  with  this 
mid-point  time.  This  cost  has  tha  probabil- 
ity pi  " .20  associated  with  it.  Hake  e 
similar  calculation  for  the  othar  segments, 
*2*  *3,  *od  84.  Than  celculata  the 

expected  value  and  atandard  deviation  for 
thesa  four  probabilities.  Thesa  tarns  ara 
then  edded  es  pert  of  the  sum  of  tha  othsr 
expected  values  and  tha  squares  of  tha 


lln  this  section  we  assume  thet  tha  cost  of 
all  work  performed  by  tha  client  end  tha  user 
is  included  in  the  analysis.  If  othar 
organizations  or  the  public  also  perform  work 
or  bear  eny  of  the  costs,  such  costs  must 
aleo  be  factored  into  the  analysis. 


standard  deviations  of  the  cost  terms  of  that 
year  as  dascribad  previously. 

Tha  expected  value  and  standard  deviation  of 
cost  for  each  yaar  must  then  be  incorporated 
into  a present  value  analysis  in  the  follow- 
ing way.  First,  apply  the  appropriate 
discount  rata  to  each  of  tha  yearly  expected 
valuas  of  cost.  Tha  sum  of  this  result  is 
tha  present  value  of  expected  cost.  The 
present  value  standard  deviation  is  calcu- 
lated as  follows: 

o Find  the  standard  deviation  of  cost 
for  each  year  as  the  square  root  of 
the  variance  of  cost  for  each  year. 

o Apply  tha  appropriate  discount  factor 
to  aach  year's  standard  deviation. 
This  yialds  the  present  value  of  each 
year's  standard  deviation. 

o Square  aach  of  thasa  fcctors  tc 
obtain  the  presant  value  of  aach 
year's  variance. 

o Sum  aach  of  these  present  value  vari- 
ances. 

o Take  tha  square  root  of  this  sum. 
This  is  the  standard  deviation  of 
total  cost. 


A laase-varsus-buy  calculation  can  elso  ba 
made  by  including  the  purchase  cost  of  the 
system  But  this  Bust  also  ba  handled  on  a 
probabilistic  basis.  In  the  example  of  Fig- 
ure 10,  there  is  a 100Z  chance  that  System 
A2  will  ba  required  for  the  initial 
installation.  As  seen  in  Figure  10,  there  is 
a 56X  chance  that  System  A2  will  be  naaded 
in  Year  3,  852  chance  in  Year  4,  and  1002 
chance  in  Yaar  5.  Using  this  data,  the 
probability  of  purchasing  A2  is  each  year 
(given  that  it  was  not  purchased  previously) 
esn  be  calculated.  This  data  can  be 

converted  into  an  expected  value  and  standard 
deviation  of  purchase  price  for  that  yaar  and 
these  values  added  to  tha  other  yearly 
expected  values  and  standard  deviations  as 
described  previously. 

Having  calculated  the  presant  value  of  the 
expected  value  end  standard  daviation  of 
cost,  the  PVLCC  can  ba  obtained  as  a normal 
distribution,  similar  to  Figures  5b  and  5c. 
Now  tha  factor  of  cost  risk  can  be  introduced 
in  the  seme  wey  as  schedule  risk  was  treated. 
Namely,  tha  client  should  indicate  the  cost 
risk  they  ara  willing  to  assume,  where  cost 
risk  is  defined  as  the  probability  of  cost 
overrun.  For  example,  the  expected  value  of 
cost  has  e 502  chance  of  overrun  and  this  may 
be  unsatisfactory  to  tha  client.  If  the 

client  is  only  willing  to  assume  a 102  chanca 
of  overrun,  for  axampla,  this  amount  is 
applied  to  the  cost  probabilistic  valuas,  as 
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illustrated  in  Figure  5c,  and  the  valua  of 
cost  obtained.  For  this  exaaq>le  of  a 1CZ 
chanca  of  overrun,  tha  value  of  cost  “ C-  ♦ 
1.28. 

where  Ce  “ expactad  valua  of  cost 

0“ “ standard  deviation  of  cost 

Values  of  cost  for  other  values  of  risk  nay 
ba  similarly  calculated  using  data  from  a 
standard  normal  probability  distribution. 

9.6  Consideration  of  Growth  Factors 

Here  ara  three  other  related  factors 
which  often  arise  in  an  evaluation: 

o Upward  Compatibility 

o Growth  Potential 

o Flexibility 

Here  is  a description  of  how  they  would  be 
treated  under  this  evaluation  method. 

The  first  step  ia  to  undarstand  and  dafina 
what  tha  client  maans  by  these  tarns.  Gen- 
erally the  tent  "Upward  Compatibility"  is 
used  to  connote  that  tha  system  can  be 
reconfigured  to  provide  greater  capability  by 
adding  additional  alementa.  That  is,  it  can 
be  modifiad  by  using  all  or  part  of  the 
original  system,  thus  providing  tha  greater 
capability  at  lass  cost  and  disruption  than 
if  a second,  totally  naw  system  were  used. 

"Growth  Potential"  is  quite  related  to  the 
previous  definition  of  Upward  Compatibility. 
In  this  case  the  site  of  the  job  may  be 
"growing"  or  increasing,  and  henca  a larger 
capability  may  be  needed. 

"Flexibility"  generally  means  that  the  set  of 
jobs  may  change,  and  the  client  would  like 
the  original  system  to  be  sufficiantly 
genaral-purpose  so  that  its  capabilities  are 
sufficient  to  perform  the  new  set  of  jobs 
rather  than  just  the  original  set  of  jobs. 

Thus,  all  three  of  thesa  terms  suggest  that 
the  client  has  some  other  set  of  jobs  in  mind 
besides  the  originally  defined  set  of  jobs. 
In  kaeping  with  tha  evaluation  approach 
described,  hara  is  how  these  terms  may  be 
included  in  the  evaluation.  First,  ex- 

plicitly define  a rapraaentative  set  of  other 
jobs  which  may  be  required  to  be  performed  by 
the  system.  Second,  indicate  both  the  dates 
when  these  jobs  will  be  performed  (such  as  in 
Taars  4 and  5)  and  the  probability  that  the 
jobs  will  occur.  This  may  be  a subjective 
estimate  of  the  probabilities.  Thus  both 

sets  of  workloads  now  bacome  the  total 
requirement.  And  ths  designer  is  required  to 
configure  his  system  design  to  accommodate 
the  total  set  of  jobs.  In  general,  the 
dasign  and  evaluation  approach  used  is  the 


same  one  used  in  tha  case  of  job  uncertain- 
ties (Figure  10).  That  is,  the  client  will 
validate  that  tha  total  system,  including 
changes,  is  capable  of  handling  tha  total 
workload,  with  proper  rasponse  times,  if  it 
should  occur.  In  addition,  the  client's 
evaluator  will  calculate  tha  total  cost  on  a 
probabilistic  basis  and  apply  the  cost  risk 
factor  to  estimate  tha  total  cost  to  be  used 
in  the  evaluation. 

9.7  Consideration  of  Superior  Characteristics 

The  racommeuded  evaluation  approach 
described  thus  far  can  be  sumnarized  as 
follows: 

o All  systems  have  been  designed  to 
perform  the  same  set  of  operational 
jobs  and  to  meet  all  specified 
constraints. 

o All  systems  will  become  available  at. 
tha  specified  date(s),  taking  into 
account  an  acceptable  risk  of  sched- 
ule overrun. 

o The  prefarred  system  is  the  one  which 
requires  the  lowest  cost  (PVLCC), 
taking  into  account  an  acceptable 
risk  of  cost  overrun. 

However,  sometimes  a designer  providas 
one  or  raora  characteristics  (generally  at 
greater  cost)  which  are  clearly  superior  to 
the  lowest  cost  system  alternative.  Now  the 
quastion  raised  is,  are  these  incremental 
superiorities  provided  worth  the  difference 
in  cost? 

Tha  kay  factor  to  be  analysed  is,  hava 
thesa  superior  characteristics  been  con- 
siderad  in  performing  the  oparational  jobs 
which  have  baan  evaluated?  Or  are  there 
othar  jobs  which  would  demonstrate  aach  of 
tha  suparior  system's  characteristics?  In 
tha  former  case,  a system's  suparior  char- 
acteristics may  have  already  been  accounted 
for  in  the  cost  calculations.  Hence,  no 
further  "credits"  need  be  given  to  that 
system.  In  the  latter  casa,  tha  avaluator 
can  calculate  the  additional  "credits"  to  ba 
given  as  follows: 

o Clearly  defina  all  other  jobs  to 
which  these  superior  charactaristics 
would  apply. 

o Estimate  how  much  additional  cost 
would  have  tc  ba  paid  by  the  client 
if  the  lowast  cost  system  ware  used 
for  thesa  jobs  rather  than  the 
suparior  system.  This  cost  is  obvi- 
ously a function  of  how  oftan  each 
job  is  performed  during  the  system 
lifa  cycla,  or  the  probability  of  its 
being  performed.  This  additional 
cost  should  be  added  to  tha  lowest 
cost  system  to  datermine  what  the 
true  PVLCC  would  be  for  all  systems. 
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Mote  that  what  we  have  done  is  to 
enlarge  the  set  of  operational  jobs  to  he 
done,  and  enlarged  the  total  costs  required 
to  do  them.  Thus  this  new  total  cost  can  be 
the  basis  of  the  system  selection. 

PAST  IV.  APPLYING  EVALUATION  METHOD  TO  A 
PROPOSAL 

The  previous  sections  presented  a method  for 
performing  an  architectural  design  study 
(involving  systems  analysis,  synthesis  and 
evaluation  of  alternatives)  in  an  environment 
where  there  is  close  contact  with  the  cli- 
ent. A proposal  effort  fundamentally  in- 
volves the  saae  systems  planning  functions  as 
described  for  the  architectural  study.  How- 
ever, instead  of  the  designer  synthesizing 
and  evaluating  all  of  the  system  alternatives 
and  selecting  the  preferred  one,  a set  of 
competing  designers  each  designs  a proposed 
system  and  submits  these  to  the  client 
evaluators  for  their  selection.  Here  are  the 
differences  which  make  it  more  difficult  to 
"optimize"  a system  in  a proposal  effort  than 
in  an  architectural  design  study: 

o First,  the  systea  requirements  are 
generally  in  the  form  of  technical 
specifications  with  firm  mandatory 
requirements.  This  may  force  the 
designer  to  provide  extra  capabili- 
ties if  the  off-the-shelf  entities  to 
be  employed  do  not  exactly  match  the 
mandatory  requirements. 

o The  second  and  most  important  dif- 
ference is  that  there  generally  is 
little  opportunity  to  make  contact 
with  the  client  prior  to  submission 
of  the  proposal,  and  hence  it  is  more 
difficult  to  "optimize"  the  design  in 
terms  of  the  client's  desires.  Thus 
it  is  very  important  that  the  de- 
signer review  the  Bequest  for  Pro- 
posal and  make  certain  that  he  under- 
stands what  the  client  is  requesting 
and  the  details  of  the  evaluation 
method  to  be  used.  There  should  be 
an  opportunity  for  the  designer  to 
obtain  clarification  of  any  fact 
which  is  ambiguous  to  him. 

With  these  differences  in  mind,  we  shall  now 
describe  how  to  apply  the  previous  systems 
planning  approach  to  the  client  proposal 
process. 

10.  APPLICATION  OF  METHOD  TO  A TECHNICAL 
CHARACTERISTICS  TYPE  OF  PROCUREMENT 

In  this  scenario  it  is  assumed  that  the 
client  provides  the  systea  requirements 
primarily  in  the  fora  of  technical  specifica- 
tions rather  than  operationally  oriented 
jobs.  The  systea  design  and  evaluation 
approach  now  recoiaended  will  still  be  based 


on  the  approach  previously  presented  but  with 
the  following  changes  as  indicated. 

o Technical  specifications  are  again 
presented  as  two  levels:  1)  a 

mandatory  minimum,  and  2)  desirable 
features. 

o An  additional  aid  to  the  designers 
would  'oe  the  inclusion  of  operation- 
ally oriented  information  regarding 
the  operational  use  of  the  system 
(jobs  and  functions  to  be  performed). 

o All  desirable  features  will  be  de- 
scribed, and  the  value  of  providing 
each  of  these  features  will  be  pro- 
vided to  all  designers.  These  values 
will  be  derived  from  the  architec- 
tural design  studies  which  were  per- 
formed at  some  previous  time,  and 
which  were  the  basis  of  the  technical 
specifications.  Based  on  the  archi- 
tectural studies,  the  client  should 
also  provide  the  designers  with  eval- 
uation functions  indicating  the  worth 
of  exceeding  the  mandatory  minimum 
requirements.  That  is,  what  is  the 
value  of  exceeding  a minimum  manda- 
tory requirement  in  terais  of  its 
dollar  savings  somewhere  else.  As 
described  previously,  each  of  these 
values  is  equal  to  the  lowest  addi- 
tional cost  of  performing  the  jobs 
neeaing  these  functions  (or  providing 
additional  performance)  if  the  func- 
tions (or  additional  performance) 
were  not  provided. 

o Each  designer  would  then  atteopt  to 
design  a system  exactly  meeting  each 
mandatory  requirement.  However,  the 
designer  will  also  consider  if  it  is 
possible  to  make  trade-offs  among 
related  parameters  which  will  meet  a 
joirt  requirement  at  lower  cost  than 
the  cost  of  meeting  (or  exceeding) 
the  requirements  singularly,  taking 
into  account  the  value  of  the  addi- 
tional features  or  performance. 

o Ideally,  the  client  would  provide  the 
designer  with  the  value  of  exceeding 
the  mandatory  requirements.  Each 
designer  could  then  properly  "opti- 
mize" his  proposal  in  terms  of  meet- 
ing all  requirements  at  lowest  PVLCC 
to  the  client,  taking  into  account 
the  value  of  desirable  features  as 
well  as  all  significantly  superior 
characteristics.  However,  if  the 
client  does  not  provide  these  values 
and  the  designer  finds  he  must 
include  these  "extras"  in  his  design, 
he  should  estimate  its  value  using 
the  method  described  previously. 

o In  either  case  the  client  should  also 
validate  such  calculations  and  select 
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that  designer  which  meets  all  re- 
quirements and  performs  all  jobs  at 
lowest  total  PVLCC. 

o Developmental  uncertainties  as  re- 
flected into  schedule  risk  and  cost 
risk  would  be  treated  in  the  same  way 
as  previously  described. 

PAST  V.  CONCLUSIONS 

In  this  paper  we  have  presented  several 
potential  pitfalls  which  can  occur  in  the 
process  of  designing,  evaluating  and  select- 
ing the  preferred  system  for  clients.  Some 
are  fairly  obvious;  some  are  not.  These 

pitfalls  and  other  conclusions  reached  in 
this  paper  can  be  summarized  as  follows: 

a.  Unless  the  requirements  of  the  job 
are  clearly  stated  and  understood  by  the 
designers,  they  will  not  be  able  to  design 
their  systems  appropriately.  Thus,  some 
means  should  always  be  available  for  further 
discussion  and  clarification  of  these  re- 
quirements prior  to  the  start  of  system 
design  efforts.  This  opportunity  is  gen- 
erally available  to  designers,  and  should  be 
utilized  early  in  the  design  process. 

b.  Unless  an  objective  procedure  for 
evaluating  system  alternatives  is  provided  to 
the  designers  by  the  client,  designers  will 
not  be  able  to  perform  their  cost-performance 
trade-offs  effectively  to  arrive  at  the 
system  design  preferred  by  the  client. 


c.  The  system  requirements  should  be 

stated  in  a way  that  will  enable  the  designer 
to  provide  what  is  desired  by  the  client  at 
lowest  total  cost.  In  architectural  studies 
where  the  client  desires  the  designer  to  make 
systems  engineering  trade-offs  among  the  key 
design  parameters,  it  is  preferable  that  the 
requirements  should  be  stated  as  operational  * 
jobs  to  be  done  rather  than  a set  of  detailed  I 
system  characteristics.  If  the  client  also 
wishes  to  include  a set  of  technical  char- 
acteristics as  mandatory  minimum  require- 
ments, with  additional  desirable  features,  it 
would  be  helpful  to  list  each  design  con- 
straint in  two  ways:  1)  a design  goal, 

indicating  the  client's  mandatory  minimum  j 
value  which  must  be  equalled  or  surpassed, 
and  2)  the  worth  of  exceeding  this  minimum 
value.  By  doing  this  the  designer  should  be 
permitted  to  make  appropriate  trade-offs 
among  design  parameters  and  thus  be  better 
able  to  satisfy  the  user  needs  at  lowest 
cost.  The  same  approach  should  be  used  in 
proposal  efforts. 

d.  Credible  evidence  of  the  accuracy 

and  reliability  of  the  proposed  work  plan 
should  be  provided  to  the  client  as  part  of 
the  architectural  design  study  and  proposal 
efforts.  Such  evidence  includes:  1)  per- 

formance validation  through  live  test  demon- 
strations, simulations  or  analysis;  2) 
reliability,  maintainability  data,  when 
available;  3)  schedule  analysis,  including 
critical  path  analyses;  4)  cost  analyses;  and 
5)  risk  analyses. 


